This manuscript establishes information-theoretic limitations for robustness of AI security and alignment by extending G odel's incompleteness theorem to AI. Knowing these limitations and preparing for the challenges they bring is critically important for the responsible adoption of the AI technology. Practical approaches to dealing with these challenges are provided as well. Broader implications for cognitive reasoning limitations of AI systems are also proven.

AI red teaming brings security thinking to LLM applications by probing failure modes such as prompt injection, output manipulation, and sensitive data exfiltration. While automated and curated benchmarks (e.g., JailbreakBench [1], HarmBench [2]) are increasingly used to test models and defenses, comparatively fewer studies analyze community scale behavior in the wild. We study ai_gon3_rogu3 [3], a 10 day competition with 504 registrants and 217 active players, to quantify solve dynamics, tactic stratification, and choke points across 11 challenges. We find sharp skill stratification, higher success for output manipulation than for data extraction, and strong effects of format obfuscation tactics, with multi step defenses remaining robust, among other insights.

As the social implementation of AI has been steadily progressing, research and development related to AI security has also been increasing. However, existing studies have been limited to organizing related techniques, attacks, defenses, and risks in terms of specific domains or AI elements. Thus, it extremely difficult to understand the relationships among them and how negative impacts on stakeholders are brought about. In this paper, we argue that the knowledge, technologies, and social impacts related to AI security should be holistically organized to help understand relationships among them. To this end, we first develop an AI security map that holistically organizes interrelationships among elements related to AI security as well as negative impacts on information systems and stakeholders. This map consists of the two aspects, namely the information system aspect (ISA) and the external influence aspect (EIA). The elements that AI should fulfill within information systems are classified under the ISA. The EIA includes elements that affect stakeholders as a result of AI being attacked or misused. For each element, corresponding negative impacts are identified. By referring to the AI security map, one can understand the potential negative impacts, along with their causes and countermeasures. Additionally, our map helps clarify how the negative impacts on AI-based systems relate to those on stakeholders. We show some findings newly obtained by referring to our map. We also provide several recommendations and open problems to guide future AI security communities.

The conversation around artificial intelligence (AI) often focuses on safety, transparency, accountability, alignment, and responsibility. However, AI security (i.e., the safeguarding of data, models, and pipelines from adversarial manipulation) underpins all of these efforts. This manuscript posits that AI security must be prioritized as a foundational layer. We present a hierarchical view of AI challenges, distinguishing security from safety, and argue for a security-first approach to enable trustworthy and resilient AI systems. We discuss core threat models, key attack vectors, and emerging defense mechanisms, concluding that a metric-driven approach to AI security is essential for robust AI safety, transparency, and accountability.

Autonomous Driving (AD) systems rely on AI components to make safety and correct driving decisions. Unfortunately, today's AI algorithms are known to be generally vulnerable to adversarial attacks. However, for such AI component-level vulnerabilities to be semantically impactful at the system level, it needs to address non-trivial semantic gaps both (1) from the system-level attack input spaces to those at AI component level, and (2) from AI component-level attack impacts to those at the system level. In this paper, we define such research space as semantic AI security as opposed to generic AI security. Over the past 5 years, increasingly more research works are performed to tackle such semantic AI security challenges in AD context, which has started to show an exponential growth trend. In this paper, we perform the first systematization of knowledge of such growing semantic AD AI security research space. In total, we collect and analyze 53 such papers, and systematically taxonomize them based on research aspects critical for the security field. We summarize 6 most substantial scientific gaps observed based on quantitative comparisons both vertically among existing AD AI security works and horizontally with security works from closely-related domains. With these, we are able to provide insights and potential future directions not only at the design level, but also at the research goal, methodology, and community levels. To address the most critical scientific methodology-level gap, we take the initiative to develop an open-source, uniform, and extensible system-driven evaluation platform, named PASS, for the semantic AD AI security research community. We also use our implemented platform prototype to showcase the capabilities and benefits of such a platform using representative semantic AD AI attacks.

We propose a novel approach that introduces three metrics: System Complexity Index (SCI), Lyapunov Exponent for AI Stability (LEAIS), and Nash Equilibrium Robustness (NER). SCI quantifies the inherent complexity of an AI system, LEAIS captures its stability and sensitivity to perturbations, and NER evaluates its strategic robustness against adversarial manipulation. Through comparative analysis, we demonstrate the advantages of our framework over existing techniques. We discuss the theoretical and practical implications, potential applications, limitations, and future research directions. Our work contributes to the development of secure and trustworthy AI technologies by providing a holistic, theoretically grounded approach to AI security evaluation. As AI continues to advance, prioritising and advancing AI security through interdisciplinary collaboration is crucial to ensure its responsible deployment for the benefit of society.

Recent works have identified a gap between research and practice in artificial intelligence security: threats studied in academia do not always reflect the practical use and security risks of AI. For example, while models are often studied in isolation, they form part of larger ML pipelines in practice. Recent works also brought forward that adversarial manipulations introduced by academic attacks are impractical. We take a first step towards describing the full extent of this disparity. To this end, we revisit the threat models of the six most studied attacks in AI security research and match them to AI usage in practice via a survey with \textbf{271} industrial practitioners. On the one hand, we find that all existing threat models are indeed applicable. On the other hand, there are significant mismatches: research is often too generous with the attacker, assuming access to information not frequently available in real-world settings. Our paper is thus a call for action to study more practical threat models in artificial intelligence security.

The National Security Agency (NSA) is starting a dedicated artificial intelligence security center, as reported by AP. This move comes after the government has begun to increasingly rely on AI, integrating multiple algorithms into defense and intelligence systems. The security center will work to protect these systems from theft and sabotage, in addition to safeguarding the country from external AI-based threats. The NSA's recent move toward AI security was announced Thursday by outgoing director General Paul Nakasone. He says that the division will operate underneath the umbrella of the pre-existing Cybersecurity Collaboration Center.

One example is the Ethics Guidelines for Trustworthy AI, from the High-Level Expert Group on AI set up by the European Commission. According to the EC guidelines, trustworthy AI should be lawful, ethical and robust [6]. The security of AI models is essential to addressing many of its requirement areas, which are becoming codified into laws and regulations, e.g., the EU AI Act [5]. As we continue to develop and rely on AI, we must prioritize security and work to address the challenges of AI safety. The market for AI startups has exploded in recent years, with many companies working on new and innovative applications. Expertise in security is not a given among all those working in AI, which makes it essential to have a dedicated focus on it to ensure safe and secure AI systems. The other day we came across this article titled "Computer security checklist for non-security technology professionals."

Late last month, Australia's leading scientists, researchers and businesspeople came together for the inaugural Australian Defence Science, Technology and Research Summit (ADSTAR), hosted by the Defence Department's Science and Technology Group. In a demonstration of Australia's commitment to partnerships that would make our non-allied adversaries flinch, Chief Defence Scientist Tanya Monro was joined by representatives from each of the Five Eyes partners, as well as Japan, Singapore and South Korea. Two streams focusing on artificial intelligence were dedicated to research and applications in the defence context. A friend who works in cybersecurity asked me this. In the world of information security, social engineering is the game of manipulating people into divulging information that can be used in a cyberattack or scam.